High-speed video from camera arrays

ABSTRACT

Methods, apparatus, systems, and articles of manufacture to process high-speed video are disclosed. An example apparatus comprises at least one memory; instructions in the apparatus; and processor circuitry to execute the instructions to: interleave a plurality of images from a first camera and a second camera in chronological order, the first camera to capture first images of the plurality of images, the second camera to capture second images of the plurality of images at a different time offset than the first camera; and generate video with an adaptive frame rate from the plurality of images by applying a mask to the plurality of images.

BACKGROUND ART

Slow motion video is generally known as a video where time appears to beslowed down. Slow motion video is often rendered using image framescaptured at a high frame rate. The slow motion video is realized bycapturing a video at a much higher frame rate compared to the frame rateat which the video will be played back. When replayed at normal speed,time appears to be moving more slowly. Accordingly, video capture forslow motion videos typically uses image capture devices capable of imagecapture at a high frame rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an electronic device that enableshigh-speed video capture;

FIG. 2A is an illustration of image capture by a single camera;

FIG. 2B is an illustration of image capture by a camera array;

FIG. 2C is an illustration of image capture by a camera array andpost-processing;

FIG. 3 is a process flow diagram of a method for high-speed video from acamera array;

FIG. 4 is an illustration of a camera array with input and outputaccording to the present techniques;

FIG. 5 is a process flow diagram of a method for high-speed video from acamera array;

FIG. 6A is an example of calculating flow based disparities;

FIG. 6B is an example of cascading backward flows to calculatedisparities;

FIG. 6C is an example of estimating a forward flow;

FIG. 6D is an example of estimating a disparity map;

FIG. 7 is an illustration of image capture by a camera array with anon-uniform frame rate; and

FIG. 8 is a block diagram showing a medium that contains logic forenabling a high-speed video capture using a camera array.

The same numbers are used throughout the disclosure and the figures toreference like components and features. Numbers in the 100 series referto features originally found in FIG. 1; numbers in the 200 series referto features originally found in FIG. 2; and so on.

Description of the Aspects

As discussed above, slow motion videos are realized by capturing videosat a high frame rate. In particular, a high-speed video is acquired,where the high-speed video is a video captured at a very high framerate. As used herein, a high frame rate may refer to a frame rate largerthan the typical 30-60 frames per second (fps). When playing thecaptured video at frame rate lower that the high frame rate, the resultis a slow motion “slo mo” video. Such slow motion videos enable detailsthat are typically missed in real life to be detected. These details areusually not seen in a high-speed video played at its original high framerate. Typically, slow motion videos are captured by a specialized camerathat can function at such high-speed rates. Camera arrays constitute apopular configuration of cameras and have their own merit for bringing aclass of new applications due to having multiple cameras in one device.

Embodiments of the present techniques enable high-speed video capturedby a camera array. In embodiments, an algorithm leverages themultiplicity of the cameras to scale up the frame rate of eachindividual camera and generate a video at a frame rate that is multipleof the individual camera frame rates. Hence, in the case of low framerate individual cameras, camera arrays can reach high-speed rates. Evenif the individual cameras are already at high frame rate, the presenttechniques can multiply this rate and enable more applications that werenot possible at the original rate.

Some embodiments may be implemented in one or a combination of hardware,firmware, and software. Further, some embodiments may also beimplemented as instructions stored on a machine-readable medium, whichmay be read and executed by a computing platform to perform theoperations described herein. A machine-readable medium may include anymechanism for storing or transmitting information in a form readable bya machine, e.g., a computer. For example, a machine-readable medium mayinclude read only memory (ROM); random access memory (RAM); magneticdisk storage media; optical storage media; flash memory devices; orelectrical, optical, acoustical or other form of propagated signals,e.g., carrier waves, infrared signals, digital signals, or theinterfaces that transmit and/or receive signals, among others.

An embodiment is an implementation or example. Reference in thespecification to “an embodiment,” “one embodiment,” “some embodiments,”“various embodiments,” or “other embodiments” means that a particularfeature, structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments, of the present techniques. The variousappearances of “an embodiment,” “one embodiment,” or “some embodiments”are not necessarily all referring to the same embodiments. Elements oraspects from an embodiment can be combined with elements or aspects ofanother embodiment.

Not all components, features, structures, characteristics, etc.described and illustrated herein need be included in a particularembodiment or embodiments. If the specification states a component,feature, structure, or characteristic “may”, “might”, “can” or “could”be included, for example, that particular component, feature, structure,or characteristic is not required to be included. If the specificationor claim refers to “a” or “an” element, that does not mean there is onlyone of the element. If the specification or claims refer to “anadditional” element, that does not preclude there being more than one ofthe additional element.

It is to be noted that, although some embodiments have been described inreference to particular implementations, other implementations arepossible according to some embodiments. Additionally, the arrangementand/or order of circuit elements or other features illustrated in thedrawings and/or described herein need not be arranged in the particularway illustrated and described. Many other arrangements are possibleaccording to some embodiments.

In each system shown in a figure, the elements in some cases may eachhave a same reference number or a different reference number to suggestthat the elements represented could be different and/or similar.However, an element may be flexible enough to have differentimplementations and work with some or all of the systems shown ordescribed herein. The various elements shown in the figures may be thesame or different. Which one is referred to as a first element and whichis called a second element is arbitrary.

FIG. 1 is a block diagram of an electronic device that enableshigh-speed video capture. The electronic device 100 may be, for example,a laptop computer, tablet computer, mobile phone, smart phone, or awearable device, among others. The electronic device 100 may include acentral processing unit (CPU) 102 that is configured to execute storedinstructions, as well as a memory device 104 that stores instructionsthat are executable by the CPU 102. The CPU may be coupled to the memorydevice 104 by a bus 106. Additionally, the CPU 102 can be a single coreprocessor, a multi-core processor, a computing cluster, or any number ofother configurations. Furthermore, the electronic device 100 may includemore than one CPU 102. The memory device 104 can include random accessmemory (RAM), read only memory (ROM), flash memory, or any othersuitable memory systems. For example, the memory device 104 may includedynamic random access memory (DRAM).

The electronic device 100 also includes a graphics processing unit (GPU)108. As shown, the CPU 102 can be coupled through the bus 106 to the GPU108. The GPU 108 can be configured to perform any number of graphicsoperations within the electronic device 100. For example, the GPU 108can be configured to render or manipulate graphics images, graphicsframes, videos, or the like, to be displayed to a user of the electronicdevice 100. In some embodiments, the GPU 108 includes a number ofgraphics engines, wherein each graphics engine is configured to performspecific graphics tasks, or to execute specific types of workloads. Forexample, the GPU 108 may include an engine that processes data from acamera array.

The CPU 102 can be linked through the bus 106 to a display interface 110configured to connect the electronic device 100 to a display device 112.The display device 112 can include a display screen that is a built-incomponent of the electronic device 100. The display device 112 can alsoinclude a computer monitor, television, or projector, among others, thatis externally connected to the electronic device 100.

The CPU 102 can also be connected through the bus 106 to an input/output(I/O) device interface 114 configured to connect the electronic device100 to one or more I/O devices 116. The I/O devices 116 can include, forexample, a keyboard and a pointing device, wherein the pointing devicecan include a touchpad or a touchscreen, among others. The I/O devices116 can be built-in components of the electronic device 100, or can bedevices that are externally connected to the electronic device 100.

The electronic device 100 also includes an array of cameras 118 forcapturing high-speed video. In embodiments, the camera array 118 is anarray of cameras, image capture devices, image capture mechanisms, imagesensors, and the like. A camera synchronization mechanism 120 may beused to control the capture of images by the camera array 118. Inparticular, with an array of cameras, the camera synchronizationmechanism can synchronize each camera to capture images at specifictimes. The frames captured by each camera can be interleaved orcomposited in temporal order, resulting in a video that samples the timedimension finer than a video captured by a single camera of the cameraarray.

The electronic device may also include a storage device 124. The storagedevice 124 is a physical memory such as a hard drive, an optical drive,a flash drive, an array of drives, or any combinations thereof. Thestorage device 124 can store user data, such as audio files, videofiles, audio/video files, and picture files, among others. The storagedevice 124 can also store programming code such as device drivers,software applications, operating systems, and the like. The programmingcode stored to the storage device 124 may be executed by the CPU 102,GPU 108, or any other processors that may be included in the electronicdevice 100.

The CPU 102 may be linked through the bus 106 to cellular hardware 126.The cellular hardware 126 may be any cellular technology, for example,the 4G standard (International Mobile Telecommunications-Advanced(IMT-Advanced) Standard promulgated by the InternationalTelecommunications Union-Radio communication Sector (ITU-R)). In thismanner, the electronic device 100 may access any network 132 withoutbeing tethered or paired to another device, where the network 132 is acellular network.

The CPU 102 may also be linked through the bus 106 to WiFi hardware 128.The WiFi hardware is hardware according to WiFi standards (standardspromulgated as Institute of Electrical and Electronics Engineers' (IEEE)802.11 standards). The WiFi hardware 128 enables the electronic device100 to connect to the Internet using the Transmission Control Protocoland the Internet Protocol (TCP/IP), where the network 132 is theInternet. Accordingly, the electronic device 100 can enable end-to-endconnectivity with the Internet by addressing, routing, transmitting, andreceiving data according to the TCP/IP protocol without the use ofanother device. Additionally, a Bluetooth Interface 130 may be coupledto the CPU 102 through the bus 106. The Bluetooth Interface 130 is aninterface according to Bluetooth networks (based on the Bluetoothstandard promulgated by the Bluetooth Special Interest Group). TheBluetooth Interface 130 enables the electronic device 100 to be pairedwith other Bluetooth enabled devices through a personal area network(PAN). Accordingly, the network 132 may be a PAN. Examples of Bluetoothenabled devices include a laptop computer, desktop computer, ultrabook,tablet computer, mobile device, or server, among others.

The block diagram of FIG. 1 is not intended to indicate that theelectronic device 100 is to include all of the components shown inFIG. 1. Rather, the computing system 100 can include fewer or additionalcomponents not illustrated in FIG. 1 (e.g., sensors, power managementintegrated circuits, additional network interfaces, etc.). Theelectronic device 100 may include any number of additional componentsnot shown in FIG. 1, depending on the details of the specificimplementation. Furthermore, any of the functionalities of the CPU 102may be partially, or entirely, implemented in hardware and/or in aprocessor. For example, the functionality may be implemented with anapplication specific integrated circuit, in logic implemented in aprocessor, in logic implemented in a specialized graphics processingunit, or in any other device.

The high-speed video capture as described herein may be used in avariety of situations. For example, slow motion videos may be capturedand used for application in sports, biology, photography, testing andmanufacturing, among others. In sports, slow motion shots are typicallyused to emphasize special moments and possibly resolve uncertainties,e.g. violations such as offside in soccer. Slow motion videos may alsohighlight the dynamics of athletes' movements and can enhance coachingthem. Another application is videography, where photographers can getslow motion footage of scenes or events. High-speed video technology isalso used in science. For example, scientists have discovered the motionmechanism of the sea snail using high-speed video and flow-trackingsystems. Moreover, high-speed video is also applied in manufacturing andtesting, such as for monitoring how a car crashes, or how concretecracks under pressure.

Devices that can capture high-speed videos are often specialized camerasthat target niche applications. In some cases, smart phones may be usedto capture slow motion videos, however video resolution is lowered toenable the slow motion videos. Camera arrays are desirable due to thecreative features enabled by camera arrays, such as computing depth,refocusing images after capture, and creating panoramas. Due to thepowerful applications they enable, camera arrays are making their wayinto mainstream devices.

With more than one camera, the present techniques enable the video framerate to be scaled up beyond the individual frame rate of each camera andreach high-speed frame rates. By synchronizing the cameras so that eachone starts capturing video at a different offset in time, then stackingthe frames in the correct chronological order (even though the framesare from different cameras), a video is obtained with an effective framerate that is multiple of the individual frame rate of each camera. In acamera array, the cameras may capture the scene from differentperspectives. This results in frames not well aligned with respect tothe scene, and a video that is jittery. According to the presenttechniques, the frames from each camera are processed so that the finalhigh-speed video is visually acceptable. Accordingly, the presenttechniques enable an algorithm to generate a high-speed video fromcamera arrays with good perceptual quality.

FIG. 2A is an illustration of image capture 200A by a single camera 202.The single camera may capture images at a frame rate along a time line250A. In particular, image capture may occur at t0 204A, t1 204B, t2204C, and t3 204D. The frame rate of the camera 202 as F, where F is anyframe rate such as 30 frames per second (fps), 60 fps, and so on.

FIG. 2B is an illustration of image capture 200B by a camera array 220.The camera array 202 may include N cameras. In the example of FIG. 2B,N=4. Specifically, the camera array 220 includes a camera 202, a camera206, a camera 208, and a camera 210. Each camera may capture images at aframe rate F. For ease of description, in the example of FIG. 2B, theframe rate is described as equal or the same for each camera. However,the cameras of the camera array 220 may capture images at differentframe rates.

Given a camera array of N cameras, C1, C2, . . . CN, the image capturecan be synchronized such that image capture among the camera arrayoccurs at an offset. For example, in FIG. 2, image capture by camera 202may occur at times t0 204A, t4 204B, t8 204C, and t12 204D, similar tothe capture described in FIG. 2A. Image capture by camera 206 may occurat times t1 212A, t5 212B, and t9 212C. Image capture by camera 208 mayoccur at times t2 214A, t6 214B, and t10 212C. Finally, image capture bycamera 210 may occur at times t3 216A, t7 216B, and t11 216C. Asillustrated, image capture occurs in time order, beginning with t0 204A,t1 212A, t2 214A and so on through t12 204D.

Accordingly, image capture can be synchronized to start capturing attimes t1, t2, . . . , to tM, where M is a total number of capturedframes. By stacking or interleaving the frames from all cameras of thecamera array in the correct temporal order, the result is a video thatsamples the time dimension much finer, actually M times finer, than thevideo captured by one single camera as illustrated in FIG. 2A. Theeffective frame rate of such a video is as follows:

Effective  frame  rate = N × F  fps

where N is the number of cameras, and F is the frame rate of eachindividual camera. In the case where each camera of the camera array hasa different frame rate, the frame rates of each camera of the cameraarray can be summed to obtain an upper bound on the effective frame rateof the interleaved video.

FIG. 2C is an illustration of image capture 200C by a camera array 220and post-processing. A video that is formed by interleaving frames fromall cameras in a camera array 220 may be jittery as a result of thedifferent perspective observed by each camera, as illustrated along thetime line 250A. In some cases, the raw interleaved video along the timeline 250A may be very jittery and completely inadequate to view. Inembodiments, a post-processing algorithm is applied to the raw video tosmooth the video into a visually plausible high-speed video, asillustrated along a time line 250B. The post processing may include viewsynthesis. In view synthesis, a virtual camera 218 with a singleperspective is selected. For every captured frame, in the rawinterleaved video, a new view is synthesized from the virtual camera218. The results are illustrated at the time line 250B, where each ofthe images captured at times t0 204A, t1 212A, t2 214A and so on throught12 204D have been processed so that they appear from the perspective ofthe virtual camera 218. Put another way, all the synthesized views willbe from the perspective of the virtual camera 218. As a result, thevideo will be smooth and will not have the jitter.

Generally, depending on the view synthesis method used, additional postprocessing may be needed to reduce any potential remaining artifacts,such as inpainting or compositing. Also, note that the presenttechniques do not require the camera array to be static, and the virtualcamera may also be moving if the camera array is moving. This may occurin, for example, a handheld usage of the device including the cameraarray. Assuming, without loss of generality, that each camera cancapture video at F fps for simplicity of presentation, the effectiveframe rate of the resulting high-speed video is NxF fps.

Existing high-speed video capture solutions rely on a single camera thatcan capture images at high-speed frame rate. However, these cameras aretypically expensive and often trade off spatial resolution to raise theframe rate. The present techniques leverage the multiplicity of camerasin a camera array to offer a video with frame rate N times larger thanthe frame rate of a single camera. In some cases, the multiplicity ofcameras in a camera array may offer a video with frame rate that is thesum of all cameras in the camera array.

While the present techniques have been described in the context ofhigh-speed video capture, the camera arrays may also be used in otherapplications, including depth-enabled applications for example. Theframe rate of each camera in the array may not be high-speed due to costconsiderations, but with the present techniques, high-speed video can beobtained from camera arrays without losing spatial resolution and whilekeeping the other features, such as depth capabilities. Even if theindividual cameras have a high frame rate, the present techniques enablescaling up this rate to be even higher. Thus, a camera array consistingof a plurality of high-speed cameras may be configured to capture imagessuch that the effective frame rate is a multiple or sum of the camerasin the high-speed camera array.

In some embodiments, a camera array may be positioned such that eachcamera of the array has the same perspective. For example, a cameraarray may include four cameras pointing inwards and one lens. In thisexample, the cameras may capture video at interleaving times as in FIG.2B, but they are physically arranged so that each camera of the cameraarray is to capture the same perspective. While there is no need for analgorithm to align the frames when the cameras share the sameperspective, this configuration is very specific for high-speed videoonly. It fails to provide any of the wider applications that usuallycome with camera arrays, such as depth, post capture photography (e.g.changing the focal plane or the aperture), and lightfield applications.Generally, most camera arrays purposefully have different perspectivefor every camera to get these capabilities. Accordingly, the presenttechniques enable high-speed video using camera arrays along withadditional camera array specific applications.

FIG. 3 is a process flow diagram of a method 300 for high-speed videofrom a camera array. At block 302, images are captured by a cameraarray. The cameras C1, C2, . . . CN of the camera array may be used tocapture frames, where the timing of each capture is staggered or offset.Each frame may be placed in the in the correct temporal, chronologicalorder. At block 304, view synthesis is performed. For every input frame,a new frame is synthesized from the perspective of the virtual cameraposition.

At block 306, post-processing is performed to generate a smooth videofrom the virtual camera. Post processing the synthesized views willremove any remaining artifacts. Generally, view synthesis algorithmsaccording to block 304 will address regions around depth boundaries ofclose objects with large gaps in depth. By synthesizing the new views,any holes or occlusions may occur. Specifically, some regions are proneto artifacts due to large dis-occlusion when changing the perspective inthe new synthesized view. For example, some view synthesis techniquesmay generate holes in the synthesized frames in these areas. For theholes, a post processing step of hole filling/inpainting is needed.Moreover, since the frames are from different cameras, color mapping maybe used to reconcile the differences between their properties and havesmoother looking video. As used herein, color mapping is a function thatmaps or transforms the colors of one image to the colors of anothertarget image. In embodiments, color mapping may be referred to as atechnique that results in the mapping function or the algorithm thattransforms the image colors. A hole may refer to a break ordiscontinuity in the texture, mesh, or geometry of an image. Holefilling is a process of calculating, extracting, or determiningappropriate values for the pixels or regions where holes appear.Similarly, inpainting may refer to the application of algorithms toreplace lost or corrupted parts of the image data. In some cases,inpainting may be referred to as image interpolation or videointerpolation. Other potential post processing steps are imagecompositing, as further described below.

FIG. 4 is an illustration of a camera array 400 with input and outputaccording to the present techniques. In FIG. 4, N=4, and time increasesaccording to the time line 450. Input from the system includes frames414, 416, 418, and 420 from four cameras 402, 404, 406, and 410,respectively, at interleaving times. The output includes frames 422 fromvirtual camera 412 with a speedup in frame rate by a factor of four.

Each of the frames 414, 416, 418, and 420 are used to synthesize frames422 with synthesized camera views from the virtual camera 412. To ensurethe best possible quality of the synthesized views, the virtual cameraposition is selected to be as close as possible to all cameras of thecamera array. Accordingly, physical location of the virtual camera 412may be at the center of the camera array 400, where the camera arrayincludes cameras in a 2×2 grid. In this example, if each camera has aframe rate of 30 fps, the effective frame rate of the virtual camera is4×30=120 fps. While a grid or rectangular layer of cameras has beendescribed, the camera array as described herein may have any layout,such as circular, linear, triangular, or any other polygon ormulti-linear layout.

Although particular techniques are described herein, many differentalgorithms may be used for view synthesis and post-processing, such ashole filling or inpainting. The present techniques are not limited to aspecific view synthesis or post processing technique. In embodiments,content aware warping may be used for view synthesis, while image/videocompositing is used for post processing.

FIG. 5 is a process flow diagram of a method 500 for high-speed videofrom a camera array. At block 502, raw camera fames are obtained fromthe camera array. In embodiments, the images are obtained from eachcamera in a round-robin fashion, thereby placing the frames in thecorrect temporal order. The frames may also be sent to a buffer as theyare captured at an offset. Additionally, in embodiments, a controllermay be used to ensure the frames are obtained and placed in the correctorder.

At block 504, view synthesis is performed. For every frame, a new viewis synthesized from a virtual camera position. In some cases, contentaware warping is performed. In embodiments, content-aware warping is afast technique to synthesize new views with a pleasing, plausibleperceptual quality. As used herein, a plausible perceptual qualityrefers to the series of frames having little to no discernable jumps,jitters, or other artifacts. The regions around depth boundaries withlarge gaps in depth will not have holes, as this method stretches thetexture in the regions to respond to disocclusion due to perspectivechange, hence implicitly filling the holes. When looking at eachindividual frame, it will look perceptually acceptable and the implicithole filling is sufficient. On the other hand, the resulting video ofinterleaved synthesized views will be much better aligned than the rawinterleaved frames, but it will still need some post processing. Indeed,in this intermediate resulting video, flickering may occur around thedisocclusion regions surrounding objects that are static or very slowlymoving, close to the camera, and/or with big gap depth from theirbackground.

While stretching the texture is sufficient for good visual quality inindividual frames coming from C1, C2 . . . CN, there is no guaranteethat this implicit hole filling is coherent across temporallyconsecutive frames coming from different cameras Ci and Cj. Theflickering is caused by this mismatch in the regions around disocclusionareas in temporally adjacent frames. To fix this problem videocompositing is used, which will also be more efficient for anycompression.

Thus, view synthesis component synthesizes the new views from the newperspective. Content-aware warping processes the input target disparityor correspondences, which are often noisy and outputs a smooth/cureddisparity, more appropriate to render perceptually pleasant view fromthe target virtual camera. In embodiments, target disparity may refer toa difference in image coordinates between a pair of frames, where eachframe comes from a different camera pose. Point correspondences mayrefer to feature points in a pair of frames that occur due to the sameobject in the real world appearing at different places when capturedfrom different camera positions. In content aware warping, the imagewith disparity or the image with point correspondences to the target areused to synthesize new views from the new perspectives. Using a densedisparity map or sparse point correspondences is a design knob thattrades off quality versus complexity.

In embodiments, content aware warping processes the image and eitherdisparity to the target (dense) or the point correspondences to thetarget (sparse) to determine the view from another perspective. Theoutput of content aware warping may be a synthesized view from thetarget virtual camera. The output disparity does not need to cover thewhole support of the image. For example, the disparity can be on aregular grid of vertices, which is a sample of the original dense imagepixels positions. Solving for this sampled disparity is anotherparameter for a system designer to control the size of the problem, andhence balance computational complexity and memory requirements togetherwith perceptual quality requirements.

To determine the sampled disparity, a cost function is formulated interms of the desired disparity and minimize it as in Equation (1). Thesolution to this optimization problem is the sought disparity, which canbe readily used to generate the synthetic view.

E(d ₁ ,d ₂ , . . . ,d _(N))=E _(d)(d ₁ ,d ₂ , . . . ,d _(N))+αE _(s)(d ₁,d ₂ , . . . ,d _(N))  Eqn. 1

where the data and the distortion terms are defined as

$\begin{matrix}{{E_{d}\left( {d_{1},d_{2},\ldots\mspace{14mu},d_{N}} \right)} = {\sum\limits_{{polygon}\mspace{14mu} p}^{\;}\;{\sum\limits_{n}^{\;}\;\left( {o_{n} - {w_{n}^{t}d^{p,t}}} \right)^{2}}}} & {{Eqn}.\mspace{14mu} 2} \\{{E_{s}\left( {d_{1},d_{2},\ldots\mspace{14mu},d_{N}} \right)} = {\sum\limits_{{polygon}\mspace{14mu} p}^{\;}{f\left( d^{p,t} \right)}}} & {{Eqn}.\mspace{14mu} 3}\end{matrix}$

Note that the target disparities values are d₁, d₂, . . . , d_(N) andthe original values are o₁, o₂, . . . , o_(M). Additionally, d^(p,t) isthe vector containing the unknown values at vertices of polygon p of thegrid. Variable on is any point in polygon p and w_(n) is the vector withthe interpolation weights assigned to each vertex of polygon p for pointon. For example, w_(n) can be bilinear interpolation weights. Termf(d^(p,t)) is the perceptual distortion measure. One possible distortionmeasure is the sum of squared differences between pairs d_(i) ^(p,t) andd_(j) ^(p,t) of disparities values at the vertices of the polygon p. Thedata term constrains the desired disparities/depth d₁, d₂, . . . , d_(N)to be close/similar to the input disparity. At the same time, thedistortion term enforces the perceptual constraints on d₁, d₂, . . . ,d_(N) so that the generated view is acceptable perceptually. Manyvariations of the Equation (1) are possible depending on the applicationitself. For example, variants of Equation (1) include respectively forvideo stabilization, synthesizing views for autostereoscopic displays,and generating cinematographs.

In some cases, directly capturing RGBD data can result in a tradeoffwith the high-speed speedup described above. For every frame captured byC1, C2 . . . and CN, content-aware warping requires as input both theRGB information and some depth information. This depth information maybe in the form of a dense target disparity map or sparse pointcorrespondences to the virtual camera. It is possible to capture RGBDdata from the camera array for use as input with content aware warping.To obtain RGBD data, N cameras are grouped into N/2 stereo pairs and thedepth is extracted. However, when using depth for the content awarewarping, the resulting high-speed video will have a frame rate N/2×Ffps, or half the increase frame rate when compare to the effective framerate described with respect to FIGS. 2-4. Instead of sacrificing halfthe cameras for RGBD capture resulting in a lower effective frame rate,the present techniques can use all N cameras for high-speed capture andthen estimate the target disparity map or point correspondences, andobtain the full speedup of factor N.

In embodiments, depth may be obtained by cascading tracking trajectoriesor flow maps. Since the camera arrays capture RGB images and not RGBDimages, the frames from each of C1, C2 . . . and CN can be tracked.Depending on desired balance of quality versus computational complexity,the tracking can be either sparse or dense. Sparse tracking may use aKanade-Lucas-Tomasi features tracker. In embodiments, dense tracking maybe accomplished through the use of dense flow maps between consecutiveframes. In the examples that follow, dense flow maps without loss ofgenerality may be used for ease of description. However, the sametechniques are applicable to sparse tracking.

In dense tracking, a forward flow is computed to the next frame andbackward flow to the previous frame is computed as well. The flows arecascaded and used for interpolation to estimate the disparity from everyframe to its target “to-be-synthesized” frame from the virtual camera.Again, the virtual camera is selected to be as close as possible to allthe cameras, at the center of the camera array.

FIG. 6A is an example of calculating flow based disparities. In theexample of FIG. 6A, all the cameras 602, 604, 606, and 608 of a cameraarray 610 are used for high-speed video capture according to the presenttechniques. Accordingly, the cameras 602, 604, 606, and 608 of a cameraarray 610 are capturing images or frames at staggered, offset, orinterleaved times. Hence at any capturing instance, only one of thecameras captures a frame and all the remaining cameras are inactive. Forexample, in FIG. 6A, at a time t3 656, only camera 608 capture an image,where the image is frame F3. None of the other cameras captures at thistime t3 656, including camera 602. A disparity map should be estimatedframe F3 to F′ 660, where F′ is a corresponding synthesized view fromthe same perspective as camera 602, i.e., the hypothetical frame thatwould have been captured by camera 602 at the same time t3 656.

FIG. 6B is an example of cascading backward flows to calculatedisparities. The backward flows are cascaded as F4 to F3 620A, F3 to F2620B, F2 to F1 620C, and F1 to F0 620D to estimate the flow F4 to F0.The cascading is done by accumulating the flow and forward mapping usingbilinear interpolation. As a result, a pair of frames representing depthis found for each flow. FIG. 6B illustrates depth representations 622A,622B, 622C, and 622D for each flow, resulting in an estimated flow fromF4 to F0.

This flow from F4 to F0 is scaled by c/N. In the example of FIGS. 6A and6B, N=4 (N=4 cameras) and the scaling factor is 0.25 to estimate theflow from F4 to F′ 660. This scaling factor is based on time intervalsbetween each image capture. The finest time interval between imagecapture using the camera array 610 is t2−t1, which is the same as t3−t2,t4−t3, etc. Thus, the time interval between each image capture is equal,and can be referred to as the unit time interval. The variable c mayrepresent the number of unit time intervals between frame F4 and targetframe F′, which is equal to 1 in this case, as the separation between F4658 and F′ 660 is t5−t4.

Generally, the scalar is c/N, where 1≤c≤N−1, depending on the targettime t. In the example of FIGS. 6A-6D, this results in the followingcalculation:

$\frac{c}{N} = {\frac{{t4} - {t\; 3}}{{t4} - {t0}} = {\frac{1}{4} = {{0.2}5}}}$

where t4−t3 is the unit time interval between t5 and t4, and t5−t1 isthe total number of time intervals between t4 and t0. The scaling isjustified since F0 and F4 are temporally adjacent frames, each takenfrom the same camera 602. In embodiments, the scaling factor leveragesthe known relationship between the change in pixels from frame F0 at t0650 to frame F4 at t4 658 to the hypothetical frame F′. Because of thisrelationship between frames F0 and F4, the motion between frames F0 andF4 can be approximated as linear. To obtain the estimated flow F3 to F′,a forward flow 624 is cascaded from F3 to F4, and a backward flow 626from F4 to F′ 660 is estimated in FIG. 6C. The result is the estimateddisparity map 670 from using the desired flow 628 from frame F3 to F′ inFIG. 6D.

As described by FIGS. 6B-6D, the disparity maps can be estimated fromframe F3, the frame captured at time t3 656 by camera 608, to thehypothetical frames that would have been captured by all the remainingN−1 cameras, in this case 3 cameras: camera 602, camera 604, and camera606. At this point, the desired target disparity map can be estimatedfrom frame F3 to the virtual camera position as a weighted sum of allthe estimated flows, using the weight 1/N. Using this weight, the targetvirtual camera is equidistant from each camera of the camera array. Inembodiments, the virtual camera is located in the center of the cameraarray. Once the target disparity map has been calculated, content-awarewarping can synthesize views from the virtual camera position using thesource image and the target disparity map.

Referring now to FIG. 5, at block 506, the video including thesynthesized views from the virtual camera position is composited. Theresulting video after stacking the synthesized views is much morealigned than the original raw video. The remaining artifacts are visiblein the disocclusion areas next to the static/slow regions, particularlythe disoccluded areas with big gap in depth between a region close tothe camera and its background. These artifacts are not present in themore moving regions as explained below.

Content-aware warping does not leave holes in the synthesized views. Itconveniently results in implicit inpainting in disoccluded areas. Instatic regions or very slowly moving regions, the consecutive frames areexpected to be exactly the same or very similar. However, in thehigh-speed video, consecutive frames come from different cameras and theimplicit inpainting from content-aware warping slightly differs from oneframe to the next. Due to this slight difference, human eyes will noticethis alternating filling between the frames as the high-speed videoplays, in static regions or very slowly moving regions. In significantlymoving regions, this artifact is not seen, as the consecutive frames areanyway expected to be significantly different in these regions and willnot have this alternating jittery filling.

In the case of slow motion videos, a high-speed frame rate is typicallyunnecessary in static or slow moving regions. The high frame rate isusually most desired in the highly moving regions. As described herein,a static or slow moving region areas that experience no or very littlepixel change. No change in pixel values may indicate a person or anobject that is not moving at all or very slowly moving. For example, aslow moving region may include a change in pixels every six frames asecond. A high moving region contains areas that experience high amountsof pixel change that can indicate a person or object experiencingcontinuous movement. This may include, for example, a person running ora ball thrown in the air. Such movements are typically at speeds wherechanges occur in the order of 30 fps or higher. In some cases, a changein pixels greater than a threshold across a plurality of frames may beused determine if it is a slow moving region or a high moving region.

Since the high-speed frame rate is typically unnecessary in static orslow moving regions, in embodiments, the present techniques select onecamera of the camera array as a reference camera and obtains image datafor the static/slow regions from this camera. In the final video, everyframe is comprised of the static/slow regions from single camera videoand the moving regions from the full high-speed video. While fixing thevisible distortions, the present techniques result in a video withadaptive frame rate, varying in the video based on amount of motion.Regions of the same frame may be rendered at different frame rates.

To accomplish this adaptive frame rate, in every synthesized view a maskand blending is applied to each frame. In every synthesized view, thehighly moving regions and the remaining regions are identified to obtaina mask from the synthesized view. To identify the various regions, thetemporal gradient of the video is analyzed. In embodiments, an imagegradient represents the spatial derivative of an image, indicating thebrightness variation within the image. In a sequence of images, thetemporal gradient is the temporal derivative and gives information onchanges between temporally consecutive frames, or in other words itgives information on motion. A new image may be found by integrating itstarget image gradient, which includes solving Poisson's equation.

Classical morphological image processing may be performed. Here, theclassical morphological image processing enables the temporal gradientto be cured to eliminate noise and blend connected components together.As used herein, morphological image processing includes a set ofoperators that are used to transform images according to characteristicssuch as size, shape, convexity, connectivity, and geodesic distance. Asa result of this processing, a mask is obtained that partitions thesynthesized view into highly moving regions when compared to the otherstatic\slow regions. In embodiments, if more accurate motion masks aredesired, a graph-cut based approach may be used. In the graph cutapproach, the problem of finding the masks can be formulated as solvinga graph-cut problem for the mask labels. In some cases, the graph cutapproach may result in heavier, more complex computations.

Having the masks, the final video may be generated by compositing. Foreach synthesized view and its mask, a new image is composited from tworegions: the highly moving regions and the static/slow regions. Thehighly moving regions are obtained as indicated in the mask from thesynthesized view itself. All remaining regions are then obtained,including static/slow regions, from the closest previous synthesizedframe from the reference camera. While a particular technique forcompositing has been described here, any compositing techniques may beused. For example, the present techniques can perform post processingcompositing using the classical Poisson technique.

Through the processing described thus far, the present techniquesgenerate a high-speed video that is perceptually pleasant. At the sametime, this video is highly adequate for compression, as it uses theavailable bandwidth wisely. Moreover, the resulting video effectivelyhas an adaptive frame rate, efficiently tailored to the amount of motionin the video, with low frame rate in static\slow regions and high framerate in high-speed regions. This approach also solves the artifacts dueto the impact of cameras differences. For example, the individualcameras in the camera array may have slight differences in theirsharpness level, which may cause artifacts in the static and slow movingregions as above.

The present techniques should not be limited to the specific processingdescribed above. For example, the present techniques can be used with amoving camera module. In this case, the view synthesis algorithm isapplicable as is, since the tracked flow or features makes the presenttechniques adaptable to the camera motion. When post processing with amoving camera module, for every current frame, an extra step may be usedfor content aware warping all the previous frames to exactly to the samecamera pose as the current frame to get correct compositing.

In the case of a non-uniform frame rate among the cameras of a cameraarray, sampling as described in FIGS. 2A-2D can be adjusted to match theframe rates of the cameras. For example, if the camera array had threecameras, with C2 704 and C3 706 at a frame rate F, and while camera C1702 has a frame rate of 2F, then the high-speed video is also at 4×Ffps. However, images from camera C3 706 is sampled twice more asillustrated in FIG. 7 along the timeline 750.

Overall, the present techniques provide a generic tool to scale up theframe rate for camera arrays, regardless of the frame rate of theindividual cameras (whether it is low or high), without trading offspatial resolution, or requiring specialized highly sensitive sensors.Additionally, as seen in the details, the final high-speed video wegenerate is more adequate for compression as it will have adaptive framerate, spatio-temporally varying in the video, based on the amount ofmotion. Highly moving areas will have higher frame rate, and slow/staticareas will have smaller frame rate.

FIG. 8 is a block diagram showing a medium 800 that contains logic forenabling a high-speed video from a camera array. The medium 800 may be acomputer-readable medium, including a non-transitory medium that storescode that can be accessed by a processor 802 over a computer bus 804.For example, the computer-readable medium 800 can be volatile ornon-volatile data storage device. The medium 800 can also be a logicunit, such as an Application Specific Integrated Circuit (ASIC), a FieldProgrammable Gate Array (FPGA), or an arrangement of logic gatesimplemented in one or more integrated circuits, for example.

The medium 800 may include modules 806-810 configured to perform thetechniques described herein. For example, an image capture module 806may be configured to capture images in an interleaved fashion. A viewsynthesis module 808 may be configured to synthesize a new frame forevery input interleaved frame from the perspective of the virtual cameraposition. In embodiments, view synthesis may be performed via contentaware warping. A post-processing module 810 may be configured togenerate a smooth video from the virtual camera. In embodiments, postprocessing includes compositing. In some embodiments, the modules806-810 may be modules of computer code configured to direct theoperations of the processor 802.

The block diagram of FIG. 8 is not intended to indicate that the medium800 is to include all of the components shown in FIG. 8. Further, themedium 800 may include any number of additional components not shown inFIG. 8, depending on the details of the specific implementation.

Example 1 is an apparatus for high-speed video. The apparatus includes acamera array, wherein each camera of the camera array is to capture animage at a different time offset resulting in a plurality of images; acontroller to interleave the plurality of images in chronological order;a view synthesis unit to synthesize a camera view from a virtual camerafor each image of the plurality of images; and a post-processing unit toremove any remaining artifacts from the plurality of images.

Example 2 includes the apparatus of example 1, including or excludingoptional features. In this example, post processing removes holes causedby synthesizing a camera view from a virtual camera.

Example 3 includes the apparatus of any one of examples 1 to 2,including or excluding optional features. In this example, postprocessing removes the remaining artifacts due to holes filledincoherently during view synthesis.

Example 4 includes the apparatus of any one of examples 1 to 3,including or excluding optional features. In this example, color mappingis used to reconcile color differences between each image.

Example 5 includes the apparatus of any one of examples 1 to 4,including or excluding optional features. In this example, the resultingframe rate is N times larger than the frame rate of a single camera.

Example 6 includes the apparatus of any one of examples 1 to 5,including or excluding optional features. In this example, each cameraimage has a different perspective compared to other camera images andsynthesizing a camera view from each image results in a single viewpoint.

Example 7 includes the apparatus of any one of examples 1 to 6,including or excluding optional features. In this example, the cameraarray is arranged to enable high-speed video, depth applications, andpost capture photography.

Example 8 includes the apparatus of any one of examples 1 to 7,including or excluding optional features. In this example, a frame rateof a video rendered using the plurality of images is an adaptive framerate, spatio-temporally varying in each frame, based on the amount ofmotion in regions of the synthesized camera views.

Example 9 includes the apparatus of any one of examples 1 to 8,including or excluding optional features. In this example, a perspectiveof the virtual camera is positioned at the center of the camera array.

Example 10 includes the apparatus of any one of examples 1 to 9,including or excluding optional features. In this example, a perspectiveof the virtual camera is selected to be as close as possible to allcameras of the camera array.

Example 11 includes the apparatus of any one of examples 1 to 10,including or excluding optional features. In this example, the viewsynthesis unit is to synthesize camera views via content aware warping.

Example 12 is a method for high-speed video from a camera array. Themethod includes capturing a plurality of frames using the camera array,wherein each camera of the camera array is to capture a frame at adifferent time offset; stacking the plurality of frames in chronologicalorder; synthesizing a view from a single perspective for each frame ofthe plurality of frames; and processing the synthesized camera views toremove artifacts in the images to render a slow motion video.

Example 13 includes the method of example 12, including or excludingoptional features. In this example, synthesizing the view includeshole-filling or inpainting for each frame.

Example 14 includes the method of any one of examples 12 to 13,including or excluding optional features. In this example, synthesizingthe view includes content aware warping. Optionally, content awarewarping comprises: obtaining a point correspondence between a frame andthe single perspective; calculate a disparity to minimize a sum ofsquared differences between distortion terms; generating the synthesizedview based on the disparity.

Example 15 includes the method of any one of examples 12 to 14,including or excluding optional features. In this example, synthesizingthe view includes content aware warping that synthesizes a view based ona dense disparity map or a sparse point correspondence. Optionally, thedense disparity map and the sparse point correspondence are estimated bycascading tracking trajectories or flow maps.

Example 16 includes the method of any one of examples 12 to 15,including or excluding optional features. In this example, thesynthesized camera views are processed to remove disocclusion areas.

Example 17 includes the method of any one of examples 12 to 16,including or excluding optional features. In this example, thesynthesized camera view is processed to realize an adaptive frame ratethat varies across the video according to an amount of motion.Optionally, the adaptive frame rate is achieved using a mask.Optionally, the mask is generated using morphological image processing.

Example 18 includes the method of any one of examples 12 to 17,including or excluding optional features. In this example, processingthe synthesized camera view comprises obtaining highly moving regions asindicated by a mask from the synthesized view, and obtaining allremaining regions from a closest precedent synthesized frame from areference camera.

Example 19 is a system for high speed video from a camera array. Thesystem includes a display; a camera array; a memory that is to storeinstructions and that is communicatively coupled to the camera array andthe display; and a processor communicatively coupled to the cameraarray, the display, and the memory, wherein when the processor is toexecute the instructions, the processor is to: capture a plurality offrames using the camera array, wherein each camera of the camera arrayis to capture an image at a different time offset resulting in aplurality of images; interleave the plurality of images in an order ofcapture; synthesize a view from a single perspective for each image ofthe plurality of images; and process the synthesized camera views toremove artifacts in the images.

Example 20 includes the system of example 19, including or excludingoptional features. In this example, processing the synthesized cameraviews removes holes caused by synthesizing a camera view from a virtualcamera.

Example 21 includes the system of any one of examples 19 to 20,including or excluding optional features. In this example, processingthe synthesized camera views removes the remaining artifacts due toholes filled incoherently during view synthesis.

Example 22 includes the system of any one of examples 19 to 21,including or excluding optional features. In this example, processingthe synthesized camera views comprises color mapping that is toreconcile color differences between each image.

Example 23 includes the system of any one of examples 19 to 22,including or excluding optional features. In this example, an effectiveframe rate is N times larger than a frame rate of a single camera of thecamera array.

Example 24 includes the system of any one of examples 19 to 23,including or excluding optional features. In this example, the cameraarray is arranged to enable high-speed video, depth image capture, andpost-capture photography.

Example 25 includes the system of any one of examples 19 to 24,including or excluding optional features. In this example, the framerate of the synthesized camera views is an adaptive frame rate,spatio-temporally varying across the synthesized camera views, based onthe amount of motion in regions of the synthesized camera views.

Example 26 includes the system of any one of examples 19 to 25,including or excluding optional features. In this example, a perspectiveof the synthesized camera views is positioned at a center of the cameraarray.

Example 27 includes the system of any one of examples 19 to 26,including or excluding optional features. In this example, a perspectiveof the synthesized camera views is selected to be as close as possibleto all cameras of the camera array.

Example 28 includes the system of any one of examples 19 to 27,including or excluding optional features. In this example, thesynthesized camera views are synthesized via content aware warping.

Example 29 includes the system of any one of examples 19 to 28,including or excluding optional features. In this example, a slow motionvideo is rendered by compositing the synthesized camera views.Optionally, compositing comprises obtaining a new image that comprises ahighly moving region and a static region for each synthesized view.

Example 30 is a tangible, non-transitory, computer-readable medium. Thecomputer-readable medium includes instructions that direct the processorto capture a plurality of frames using the camera array, wherein eachcamera of the camera array is to capture an image at a different timeoffset resulting in a plurality of images; stack the plurality of imagesin chronological order; synthesize a view from a single perspective foreach image of the plurality of images; and process the synthesizedcamera views to remove artifacts in the images to render a slow motionvideo.

Example 31 includes the computer-readable medium of example 30,including or excluding optional features. In this example, synthesizingthe view includes hole-filling or inpainting for each frame.

Example 32 includes the computer-readable medium of any one of examples30 to 31, including or excluding optional features. In this example,synthesizing the view includes content aware warping. Optionally,content aware warping comprises: obtaining a point correspondencebetween a frame and the single perspective; calculate a disparity tominimize a sum of squared differences between distortion terms;generating the synthesized view based on the disparity. Optionally,content aware warping synthesizes a view based on a dense disparity mapor a sparse point correspondence. Optionally, the dense disparity mapand the sparse point correspondence are estimated by cascading trackingtrajectories or flow maps.

Example 33 includes the computer-readable medium of any one of examples30 to 32, including or excluding optional features. In this example, thesynthesized camera views are processed to remove disocclusion areas.

Example 34 includes the computer-readable medium of any one of examples30 to 33, including or excluding optional features. In this example, thesynthesized camera view is processed to realize an adaptive frame ratethat varies across within each frame according to an amount of motion.Optionally, the adaptive frame rate is achieved using a mask.Optionally, the mask is generated using morphological image processing.

Example 35 includes the computer-readable medium of any one of examples30 to 34, including or excluding optional features. In this example,processing the synthesized camera view comprises obtaining highly movingregions as indicated by a mask from the synthesized view, and obtainingall remaining regions from a closest previous synthesized frame from areference camera and blending a highly moving region and a remainingregion for each frame.

Example 36 is an apparatus for high-speed video. The apparatus includesinstructions that direct the processor to a camera array, wherein eachcamera of the camera array is to capture an image at a different timeoffset resulting in a plurality of images; a means to interleave theplurality of images in chronological order; a means to synthesize acamera view to synthesize a camera view from a virtual camera for eachimage of the plurality of images; and a means to remove artifacts toremove any remaining artifacts from the plurality of images.

Example 37 includes the apparatus of example 36, including or excludingoptional features. In this example, the means to remove artifactsremoves holes caused by synthesizing a camera view from a virtualcamera.

Example 38 includes the apparatus of any one of examples 36 to 37,including or excluding optional features. In this example, the means toremove artifacts removes the remaining artifacts due to holes filledincoherently during view synthesis.

Example 39 includes the apparatus of any one of examples 36 to 38,including or excluding optional features. In this example, color mappingis used to reconcile color differences between each image.

Example 40 includes the apparatus of any one of examples 36 to 39,including or excluding optional features. In this example, the effectiveframe rate of a video rendered using the plurality of images is N timeslarger than the frame rate of a single camera.

Example 41 includes the apparatus of any one of examples 36 to 40,including or excluding optional features. In this example, each imagehas a different perspective compared to other images captured by othercameras of the camera array and synthesizing a camera view for eachimage results in a single perspective for the plurality of images.

Example 42 includes the apparatus of any one of examples 36 to 41,including or excluding optional features. In this example, the cameraarray is arranged to enable depth applications, post capturephotography, or any other application that requires a camera array.

Example 43 includes the apparatus of any one of examples 36 to 42,including or excluding optional features. In this example, a frame rateof a video rendered using the plurality of images is an adaptive framerate, spatio-temporally varying across the synthesized camera views,based on the amount of motion in regions of the synthesized cameraviews.

Example 44 includes the apparatus of any one of examples 36 to 43,including or excluding optional features. In this example, a perspectiveof the virtual camera is positioned at the center of the camera array.

Example 45 includes the apparatus of any one of examples 36 to 44,including or excluding optional features. In this example, a perspectiveof the virtual camera is selected to be as close as possible to allcameras of the camera array.

Example 46 includes the apparatus of any one of examples 36 to 25,including or excluding optional features. In this example, the means tosynthesize a camera view is to synthesize camera views via content awarewarping.

Example 47 includes the apparatus of any one of examples 36 to 46,including or excluding optional features. In this example, a video isrendered by compositing.

It is to be understood that specifics in the aforementioned examples maybe used anywhere in one or more aspects. For instance, all optionsfeatures of the computing device described above may also be implementedwith respect to either of the methods or the computer-readable mediumdescribed herein. Furthermore, although the flow diagrams and/or statediagrams may have been used herein to described aspects, the techniquesare not limited to those diagrams of to corresponding descriptionsherein. For example, flow need not move through each illustrated box orstate or in exactly the same order as illustrated and described herein.

The present techniques are not restricted to the particular detailslisted herein. Indeed, those skilled in the art having the benefit ofthis disclosure will appreciate that many other variations from theforegoing description and drawings may be made within the scope of thepresent techniques. Accordingly, it is the following claims includingany amendments thereto that define the scope of the present techniques.

RELATED APPLICATION

This patent arises from a continuation of U.S. patent application Ser.No. 15/198,727, which was filed on Jun. 30, 2016. U.S. patentapplication Ser. No. 15/198,727 is hereby incorporated herein byreference in its entirety. Priority to U.S. patent Ser. No. 15/198,727is hereby claimed.

What is claimed is:
 1. An apparatus to process high-speed video, theapparatus comprising: at least one memory; instructions in theapparatus; and processor circuitry to execute the instructions to:interleave a plurality of images from a first camera and a second camerain chronological order, the first camera to capture first images of theplurality of images, the second camera to capture second images of theplurality of images at a different time offset than the first camera;and generate video with an adaptive frame rate from the plurality ofimages by applying a mask to the plurality of images.
 2. The apparatusof claim 1, wherein the processor circuitry is to synthesize a cameraview from a virtual camera for ones of the plurality of images.
 3. Theapparatus of claim 2, wherein the processor circuitry is to executeinstructions to remove a hole caused by the synthesizing of the cameraview.
 4. The apparatus of claim 2, wherein the processor circuitry is tois to execute instructions to remove artifacts due to holes filledincoherently during the synthesizing of the camera view.
 5. Theapparatus of claim 2, wherein the first camera has a differentperspective than the second camera, the synthesizing of the camera viewresulting in a single view point.
 6. The apparatus of claim 2, whereinthe processor circuitry is to is to execute instructions to synthesizethe camera view based on content aware warping.
 7. The apparatus ofclaim 2, wherein the processor circuitry is to is to executeinstructions to select a perspective of the virtual camera based on thefirst and second cameras.
 8. The apparatus of claim 1, wherein theadaptive frame rate is larger than a frame rate of a single camera. 9.The apparatus of claim 1, wherein the first and second cameras arestructured to enable at least one of high-speed video, depthapplications, or post capture photography.
 10. A computer readablemedium comprising instructions which, when executed, cause one or moreprocessors to at least: interleave a plurality of images from a firstcamera and a second camera in chronological order, the first camera tocapture first images of the plurality of images, the second camera tocapture second images of the plurality of images at a different timeoffset than the first camera; and generate video with an adaptive framerate from the plurality of images by applying a mask to the plurality ofimages.
 11. The computer readable medium of claim 10, wherein theinstructions are to cause the one or more processors to synthesize acamera view from a virtual camera for ones of the plurality of images.12. The computer readable medium of claim 11, wherein the instructionsare to cause the one or more processors to remove a hole caused by thesynthesizing of the camera view.
 13. The computer readable medium ofclaim 11, wherein the instructions are to cause the one or moreprocessors to remove artifacts due to holes filled incoherently duringthe synthesizing of the camera view.
 14. The computer readable medium ofclaim 11, wherein the first camera has a different perspective than thesecond camera, the synthesizing of the camera view resulting in a singleview point.
 15. The computer readable medium of claim 11, wherein theinstructions are to cause the one or more processors to synthesize thecamera view based on content aware warping.
 16. The computer readablemedium of claim 11, wherein the instructions are to cause the one ormore processors to select a perspective of the virtual camera based onthe first and second cameras.
 17. The computer readable medium of claim10, wherein the adaptive frame rate is larger than a frame rate of asingle camera.
 18. The computer readable storage medium of claim 10,wherein the first and second cameras are structured to enable at leastone of high-speed video, depth applications, or post capturephotography.
 19. A method to process high-speed video, the methodcomprising: interleaving, by executing an instruction with one or moreprocessors, a plurality of images from a first camera and a secondcamera in chronological order, the first camera to capture first imagesof the plurality of images, the second camera to capture second imagesof the plurality of images at a different time offset than the firstcamera; and generating, by executing an instruction with the one or moreprocessors, video with an adaptive frame rate from the plurality ofimages by applying a mask to the plurality of images.
 20. The method ofclaim 19, further including synthesizing a camera view from a virtualcamera for ones of the plurality of images.
 21. The method of claim 20,further including removing a hole caused by the synthesizing of thecamera view.
 22. The method of claim 20, further including removingartifacts due to holes filled incoherently during the synthesizing ofthe camera view.
 23. The method of claim 20, wherein the first camerahas a different perspective than the second camera, the synthesizing ofthe camera view from the plurality of images resulting in a single viewpoint.
 24. The method of claim 20, further including synthesizing thecamera view based on content aware warping.
 25. The method of claim 20,further including selecting a perspective of the virtual camera based onthe first and second cameras.